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HIGHLIGHTS 



The Department of Education (ED) has recently completed the Title IV 
Quality Control Study Stage Two, a study of quality in the Title IV 
student financial aid programs for the 1985-86 academic year* In 
response to concerted efforts to improve program quality, progress has 
been made in reducing the relative magnitude of error since the first 
quality control measurements were made in 1978-79 • However, even with 
this reduction, hundreds of millions of dollars are being disbursed 
incorrectly to postsecondary students each year* 

The magnitude and trend in error in the Pell Grant Program is 
illustrated in Exhibit 1. The error measured in the study went beyond 
regulatory violations, and included broad delivery system processes* 
Student error is defined as misawards attributable to the submission of 
incorrect information on students* application for financial assistance* 
Institution error is defined as misawards attributable to schools using 
incorrect data in awarding, processing, or disbursing aid or for not 
collecting necessary documentation* The dollar amounts of student and 
institutional errors are mutually exclusive. 

The Stage Two study is the first such study to measure error in all 
five of the majo^ Title IV programs* Consistently high rates and 
magnitudes of error were found in each program* For example, 54 




* 1978-79 $tudy used exp0ct0d dlsbursenwnts rather than actual. 
Total program funds on which error figures were based: 
1978-79'' $1.1 bihion 
1980-31 » $2.2 billion 
1982-83 » $2.4 billion 
1985-86 ^ $3.6 billion 



EXHIBIT 1. ABSOLUTE STUDENT AND INSTITUTIONAL PELL ERROR 
AS A PERCENT OF PROGRAM EXPENDITURES 



EKLC 



percent^ of Pell Grant recipients received incorrect awards • These 
mispayments totalled $763 million nationwide, or about 21 percent of all 
Pell Grant funds for that year. Seventy-seven percent of Campus-Based 
recipients had awards determined with incorrect data and 22 percent 
received awards in excess of need totaling $265 million. In the 
Guaranteed Student Loan (GSL) program, the amount of need certified by 
institutions was incorrect in an estimated 20 percent of cases. The 
resulting cost to the government is estimated to be $338 million over the 
life of thii loars. 

There ce many reasons for these high levels of error. Student 
misreporting of application information is a major cause. For example, 
approximately one-third of all Pell recipients are estimated to have 
award errors due to their providing erroneous information on their 
applications for assistance. This may be due to confusing forms and 
processes, the use of prospective data, and intentional misreporting. 
Institutional error also continues to be a problem. For example, an 
estimated 30 percent of all Pell recipients received incorrect awards due 
to institutions using inaccurate data for such things as cost of 
attendance and enrollment status or for not collecting necessary 
information. 



^All error figures utilized a $50 tolerance. 
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This study/ like the previous studies/ used a broad definition of 
error. Error/ for the purposes of the study/ went beyond liability and 
included quality. The error figures do not imply any deliberate waste or 
fraud/ nor should they be interpreted as placing blame on either students 
or institutions. Instead/ the error figures relate the extent to which 
Title IV awards deviate from the intended distribution of awards. Much 
of the error is inherent ia the error-prone nature of the Title IV 
delivery system/ and should not be strictly interpreted as potential cost 
savings. However / this error-prone process undermines the very objective 
of increastid c^quity. 

These problems occurred despite ED corrective actions implemented 
since 19S0 to reduce error. For example/ improvements in the criteria 
and procedures for selecting and validating recipients have been made. 
AlsO/ the proportion of Pell Grant applicants whose reported adjusted 
gross income^ taxes paid/ and Social Security benefits that must be 
verified through documentation was increased. Both of these activities/ 
coupled with improved design of the student application for financial 
assistance were implemented in an effort to reduce student error. 
Additional initiatives have been instituted to reduce institutional 
error . 

The corrective actions taken to date have focused on the individual 
data or procedure level. Error reduction ^ 3S occurred in the individual 
data elements that were the target of the corrective actions. However/ 
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error in several other items or procedures has remained constant or 
increased. (For example, a major corrective action focus has been post 
hoc verification of selected application data items. Verification has 
had a limited ripple effect on data items and applications not 
selected.) Stage Two analyses indicate that even under the most generous 
assumptions, the possibility for reducing error below a certain floor 
level using these types of corrective actions is limited. A significant 
amount of residual error remains, and will remain, unless long-term 
systematic improvements are implemented. 

To make further meaningful improvements in quality, it is necessary 
to re-examine aspects of the delivery system itself and, where 
appropriate, ciiange its structure. The Title IV Qualix^y Control Project 
has identified three major areas for quality improvement; 



♦ Simplification of the delivery process including a reduction 
in the number of data elements required for needs analysis; 

♦ Institution-based quality control to place accountability and 
authority at the appropriate level to reduce error; and 

♦ Structural changes to the delivery systems to integrate 
functions and data elements among programs and to provide 
central control over decentralized delivery system activities. 



With respect to simplification , analyses conducted indicated that for 
both the Pell Grant needs analysis usea to determine the Pell Student Aid 
Index (SAI) and the Uniform Methodology (UM) formula used to determine a 
student's Expected Family Contribution (EFC) in the Campus-Based programs: 
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♦ Reduced needs analysis formulae containing only six data 
elements closely approximate the "intended" distr iout ions of 
need for Pell and Campus -Based aid and for GSL 
certification. ''Intended" distributions were determined by 
using best values of application data in the current 
long-formula needs analyses. 

♦ Most aid recipients would have minimal or no changes in their 
Pell Award/ Campus-Based need/ and GSL certification if the 
current needs analysis formulae were replaced with a reduced 
formula . 

♦ Student error wculcl decline dramatically under reduced needs 
ana lysis f o r mu 1 ae . 

One of the major findings of the Title IV QC study was that 
systematic institution-based quality control procedures are associated 
with lower rates of institutional error. For example/ schools that 
regularly reviewed a sample of recipients for error and who used either 
manual quality control checks or other auxiliary quality control 
procedures (e.g., using auditors/ using consultants/ checking other 
offices/ or interviewing students) had institutionally-caused errors for 
an estimated 19 percent of all Pell Grant recipients in 1985-86. By 
contrast/ twice as high a percentage (38.5 percent) of institutional 
error was associated with Pell Grant recipiencs at schools with little or 
no regular quality control procedures in place. This argues for expanded 
support for a quality control initiative that places responsibility for 
error management at the Institutional level. 

Finally/ the continued high error rates since 1978-79 indicate that 
structural changes to the delivery systems are needed. One product of 
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the Title IV QC Study is a paper entitled Delivery System Quality 
Improvements > This paper establishes a comprehensive framework for 
analyzing and recommending specific long-term quality improvements to the 
delivery of Title IV assistance* These quality improvements include an 
integrated needs analysis structure and decentralized/ integrated 
processors controlled through a central data base and a central 
disburser . 

In summary there is a pattern of decreasing error magnitude in the 
Title IV programs over the last decade* Much of this is probably related 
to improvements in the quality of specific data items and procedures 
targeted by corrtctive actions* There remains, however, a large amount 
of residual error, not accessible through current means* What is needed 
is long-term action targeted at delivery system quality, including the 
simplification of data and procedures, institutional level quality 
control, and improvements in the structure by which student aid is 
delivered. 
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INTRODUCTION 



Stage Two of the Title IV Quality Control Project is the first 
comprehencive evaluation of quality in the Department of Education's (ED) 
major student aid programs. As such, the purpose of the study is to 
identify, measure, and analyze error during the 1985-86 academic year in 
each of the five major Title IV programs, including the Pell Grant 
program, the Campus-Based programs (consisting of Lhe Supplemental 
Educational Opportunity Grant, National Direct Student Loan - renamed 
Perkins Loans by the reauthorization of the Higher Education Act - and 
College Work-Study programs) and the Guaranteed Student Loan (GSL) 
program. This document will briefly summarize the major components of 
the study and will discuss the methodology used, the key findings of the 
study/ and key corrective action recommendations. 

!•! STUDENT AID PROGRAMS 

Five Kiajor Federal programs of financial assistance to post-high 
school students have evolved from legislation of the 1960 's and early 
1970 's. Collectively, these five programs are known as the "Title IV" 
programs after Title IV of the Higher Education Act (20 JSC 1070a, b, 
1071, 1087aa, 2751 ; and 42 USC 2751 ) which, as amended, provides the 
legislative authority for them. 
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The Pell Grant (formerly BEOG) progr?im is an entitlement program for 
undergraduate students/ designed to be the foundation upon wh^ch all 
other Federal/ state, and private aid builds/ and was implemented in 
acad'^mic year 1973-74. Eligibility for ''he program is determined through 
a nationally uniform financial eligibility test/ called the Family 
Contribution Schedule/ developed by ED and approved by Congress every 
year. Currently/ awards range from $200 to $2/100 for full-time 
students/ and are reduced proportionately for part-time students. 

Each of the three Campus-Based programs is administered bv 
participating postsecondary institutions. Each participating institution 
applies annually for subsequent year fundS/ and reports prior year 
activity on the "Fiscal Operations Report and Application to Participate 
in Federal Student Financial Aid Programs" (FISAP). The Campus-Based 
award process perhaps best demonstrates one of the major differences 
between these programs and, for instance/ the Pell program. Individual 
institutions are free within the regulations to establish the parameters 
Within which Campus -Based aid is awarded. Financial aid administrators 
at these institutions award Campus-Based funds in conjunction with other 
programs to meet student need as determined by an ED-approved need 
analysis procedure, most often the Uniform Methodology. Campus aid 
administrators tailor awards to meet this need according to available 
funds and the institutional aid packaging philosophy. This aid packaging 
philosophy may dictate the sequence/ amount/ or type of aid given and the 
percentage of need met for different types of studencs. 
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The Guaranteed Student Loan Program (GSL) provides the most financial 
assistance to postsecondary students of all Title IV programs. It makes 
available to students atten<?ing eligible postsecondary institutions loan 
funds with which to meet educational expenses. The program uses capital 
provided through private sector banks, savings and loan associations, 
credit unions, and educational and other financial entities. The Federal 
government subsidizes these loans through "special allowances" to lenders 
in order to increase lender yields to provide a more equitable return. 
In addition, the government pays the full interest on borrowed amounts 
when students are in school, in a ''grace period," or during periods of 
deferment. To receive a Federally-subsidized GSL, students must rr.eet 
general eligibility criteria, similar to other Title IV programs, and 
also demonstrate financicil need. 

Students are responsible for repayment of loans after ceasing at 
least half time enrollment and after a brief grace period. During 
repayment, students pay both the principal and interest, while the 
Federal government continues to pay the "special allowance," the 
difference between the interest rate charged (7, 8, or 9 percent) and the 
prevailing interest rate for Treasury bills, as well as providing for a 
100 percent guarantee to lenders against default. The maximum repayment 
period is 10 years. 

1.2 QUALITY CONTROL IN STUDENT AID PROGRAMS 

A series of quality control studies have been conducted by ED in the 
^ past. However, each focused only on a single program, such as the Pell 



Grant Studies, or developed and pilot-tested a methodology for measuring 
error in one or more of the programs, which was the purpose of Stage One 
of the present study. These prior studies produced data that provide 
important reference points for the present study. 

The previous studies have continually raised issues concerning 
improving the quality of the Title IV delivery systems. Historically, 
ED'S quality strategy has principally relied upon verification of 
student-reported data items, audits and program reviews of institutions, 
conducting national studies of the magnitude and sources of error, and 
designing corrective actions aimed at individual data items. These have 
generally been mechanical solutions which have maintained the status quo. 



In recent years, however, there has been a consensus emerging that 
quality improvement strategies and tactics should focus on changes to the 
Title IV delivery system itself. These larger aspects include changes 
made to the delivery systems during recent reauthorization of the Higher 
Education Act, as well as the Institutional Quality Control Pilot Project 
(IQCPP) currently being tested by ED to assess the feasibility of 
implementing a quality control system at institutions and assessing its 
effectiveness. 



1.3 HISTORY OF ERROR IN STUDENT AID PROGRAMS 



Despite an excellent record of identifying, quantifying, and making 
management improvements to diminish error, post secondary institutions are 
still observing considerable discrepancies in the awarding of student 



1-4 



16 



financial assistance. Since the first nationwide study of error in the 
Basic Grant (now Pell Grant) Program in 1978-79/ considerable attention 
has been placed on lowering the rate and magnitude of error. Increased 
validation of student application data, forms and procedural redesigns, 
institutional quality control (QC) programs, among other activities, have 
been shown to be effective in removing some error. 

Through the grow ,h and maturation of the Title IV programs, quality 
has become an increasing concern, and will continue to be a concern of 
all those involved including Congress, ED, institutions, and students. 
The delivery systems are complex in nature and contain many aspects that 
are inherent obstacles to quality including some data items used by the 
systom to determine awards. The trend in error rates has shown that 
error continues to be significant in the Title IV programs. 

The De'*Jirtment is now faced with a critical choice. One option is to 
accept the status quo and continue to implement technical improvements to 
the student aid delivery process and accept an error rate of 15 to 20 
percent discrepantly awarded recipients. The second option is to embark 
on major, structural, changes to the delivery system in an effort to 
reduce error systematically without changing the basic intent of the 
programs themselves (i.e., equity, access, and cost-effective service). 
The current delivery process exists because of the complexities of 
providing billions of dollars to students with varying amounts and types 
of needs. Making structural changes to that delivery process will 
require serious thought, planning, and coordination so as not to disrupt 
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payments. The task before ED is very clear. In order to reduce error 
appreciably/ the current delivery system must be restructured (through 
redesigning and simplifying the current process) to design error out of 
the system and reduce the need for costly inspection. This restructuring 
can and should occur without significantly changing the intent of the 
programs/ only the manner in which they are implemented. 

1.4 TITLE IV PROJECT SUMMARY 

In January 1984. the Office of Student Financial Assistance (OSFA) of 

the U.S. Department of Education (ED) contracted with Advanced 
Technology, Inc., of Reston, Virginia, to conduct a two-stage study of 

the five principle student aid programs funded under Title IV of the 

Higher Education Act. Advanced Technology engaged Westat, Inc., of 

Rockville, Maryland. to perform field work and provide technical 
assistance in special areas such as sample design. 

Stage One of the Title IV Quality Controx , roject, conducted during 
1984, was designed as a "pilot study." The pilot study was restricted to 
the Campus-Based and GSL programs since three field studies of the Pell 
Grant program had already been done. 

The design of this study. Stage Tv/o of the Title IV Quality Control 
Project, was based on four overarching objectives: 



• To determine whether the level and patterns of error 
persist in the Title IV programs and assess any interactive 
effects . 
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To assess the effects of 
initiatives • 



prior ED corrective action 



To measure scructural error 
standard corrective actions). 



( i.e 



error not amenable to 



To describe the effects of proposed major corrective 
actions on improving quality in the delivery of Federal 
student aid. 



Stage Two is the first study to measure error across all the major 
Title IV programs, and provides an opportunity ho monitor the effects of 
corrective actions already in place or recently added to any of the 
programs as a result of recommended corrective actions from Stage One or 
the earlier Pell Stage Three Study. 




METHODOLOGY 



In this chapter we present a brief description of the methodology 
used in conducting Stage Two, including its focus, data sources, error 
definitions, sampling, data processing procedures, and analyses. 

2.1 FOCUS OF STAGE TWO 

As mentioned earlier, ti.i Stage Two study of error is the first study 
to be conducted by ED that is integrated across the Title IV programs. 
Because of this aspect of the study, many interactive effects of error, 
aad interactive analyses, can be simulated and estimated. 

Tlie basic methodology for this 5'-udy has been employed numerous times 
in previous studies of the student financial aid programs. Hence, the 
results of the current study provide an opportunit}- to compare data 
gathered from the previous Pell studies and confirm data from Stage One 
for the Campus-Based and GSL programs. Like the previous QC studies. 
Stage Two nieasured error according to a broad definition, not a more 
restricted definition agreeing with regulatory liability. Thus, the 
focus of the study was to analyze the extent and causes of incorrect 
awards and not to place blame on students or institutions. 

Data for this study were collected from three major sources: the 
institutions, the students and their parent{s), and external sources 
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which could verify data obtained from the students and their parents. 
(Exhibit 2-1 depicts these data sources graphically.) Each of these data 
sources provide important documentation for the study and are described 
briefly below. In addition, data on Pell recipients were abstracted from 
the Computed Applicant Record (CAR) maintained by the Pell central 
processor. 



• An Institutional Questionnaire (IQ) was administered to the 
financial aid ar'^ministrator during an interview at each 
sampled institution. One of its major purposes was to obtain 
information on institutional policies and procedures. This 
information was used to determine each student's correct 
need/ calculate error, identify institutional characteristics 
correlated with error/ and gather information on quality 
control practices. The IQ data were collected in February 
and March of 1986. 

• Field data collectors abstracted student data from 
institutional files during the institutional visit. Data for 
each sampled student were abstracted in the following areas: 
general eligibility. Pell Grant program data, Campus-Based 
program data, GSL program data/ documentation contained in 
the student's files, and disbursements and 
repayments/refunds. These data were abstracted from 
institutional files in February, March, and April of 1986/ 
which enabled ED to ensure that student changes, drops in 
enrollment, changes in need/ and subsequent award adjustments 
were captured and hence error data were not artificially 
inflated. The QC study methodology has shown that 
end-of-year clean-up of student files produces no significant 
changes and has virtually no affect on nationwide error 
estimates . 

• A Student Questionnaire (SQ) was administered to students 
during personal interviews, and was designed to confirm or 
verify data reported on the aid application completed by the 
student. A questionnaire was administered to parents (PQ) 
for similar purposes. However, if the dependency status of 
an independent student was confirmed in the parental 
interview^ further questions were omitted. SQ and PQ data 
were collected in March, April, and May of 1986. 

• Students and parents provided written permission for the 
release of Federal tax forms and verifying the value of 
savings and checking accounts. This permission was obtained 
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during the student and parent interviews. For a sample of 
the schools, local tax assessor's offices were contacted for 
respondents reporting home ownership, to ascertain the 
approximate market value of the respondent's home or primary 
residence. Data from secondary sources were collected in 
March, April, and May of 1986. 



2.2 ERROR DEFINITION AND MEASUREMENT 

Our approach to error definition and error measurement in Stage Two, 
as in prior studies, defined error as the difference between need or 
award calculated using data reported by the student and/or sed by ::he 
institution and the most accurate and reliable data obtained during the 
course of data collection. In the absence of such confirmatory data, for 
any given item, the value reported by the student was accepted and used 
for analysis purposes as the best value. 

Stage Two is unique in that it measures errors in all five (Pell, 
three Campus-Based programs and GSL) Title IV programs. Therefore, 
measurement of error in the Pell, Campus-Based, and GSL programs required 
the use of several conceptually distinct error definitions due to the 
unique characteristics of each program. 

Pell Error 

In the Pell program^ a change in a reported data element (e.g., AGI) 
will have a known effect on the applicant's Pell Grant award at a given 
enrollment status and cost of attendance. Thus, such changes, and 
therefore error, could be modeled precisely because Pell is a 



formula-driven, entitlement program. Pell payment error is a measure of 
differences in students' Pell Grants using actual and best data. 



Campus-Based Error 



The characteristics of the Campus-Based programs required a distinct 
and different approach to defining and measuring error. Need analysis 
performs a function much like the Pell formula. However, thj result of 
need analysis does not determine an award, but is used by institutions as 
an input to determine a student's Campus-Pased award. Thus, the effect 
that changes in student repoited data had on awards could only be 
estimated. Because of the nature of the Campus-Based programs, the 
following three error measures were required to describe the quality of 
the Campus-Based programs: 

• Campus-Based need error is a measure of the discrepancies 
in calculating students' r.eed (the difference between 
resources available and resources required to finance 
postsecondary education) . 

• Campus-Based awards in excess of need is a measure of the 
extent to which need errors caused stuuents' need to drop 
below award. Because most students have some amount of unmet 
need, not all need errors (where need drops) will cause 
awards in excess of need. This error measure approximates 
the regulatory definition of error. 

• Campus-Based distributional error converts need errors into 
likely award consequences using institutional packaging 
algorithms . Distributional error is probably the best 
overall indicator of quality in the Campus-Based programs. 

GSL Error 



Measurement of error in the GSL program presented other 
methodological problems. Institutions, one of the major foci of the 



study, play a limited role in the program: certifying the aruount for 
I'hich a student is eligible. Program limits, students who apply for 
specific amounts, and lenders and guarantee agencies jointly determine 
the actual loan amount. Often, institutions may not know the exact loan 
amount, or if the student even completed the loan process and received a 
loan. Therefore, the focus of the study was on error at the point of 
certification. (Changes made during reauthorization of the Higher 
Education Act now involve the schools directly in the disbursement of GSL 
funds.) Guaranteed Student Loan certification error is a measure of the 
overcerti f ications when best duca are substituted for reported data. 
Certifications are capped at the program lin^its of $2,500 for 
undergraduates and $5,000 for graduate students. 

Error measures in each of the Title IV programs were decomposed into 
three types of error: student reporting error (which is used to motivate 
corrective actions rather than assign responsibility), institutional 
error, and overall error. The definitions are as follows: 

• Student reporting error is the result of recipients 
providing inaccurate data at the time of application and 
subsequent to it. This decomposition is silent on whether 
the error was conscious or inadvertent or whether it was true 
at the time and subsequently changed. 

• Institutional error is the result of institutions using 
incorrect data and making errors that affect student need. 

• Overall error is the total result of incorrect student 
reporting and institutional errors in handling data, 
categorical errors, or procedural errors. 
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These errors were decomposed further to identify key individual or 
groups of errors as a basis for corrective actions analysis. Although 
all error measures are quality errors, some include true regulatory 
violations. Analysis indicates that regulatory violations are of 
significant volume. 



2 . 3 SAMPLING 



The data for this study were collected from samples of Pell Grant and 
Campus-Based recipients and GSL certifications at 297 sampled 
institutions participating in these Title IV programs. The sampling was 
conducted in two stages: 



♦ A random sample of institutions participating in the Pell 
Grant/ Campus -Based , or the GSL program was selected first. 

♦ Random samples of Pell Grant and Campus-Based recipients and 
GSL certifications were selected at each institution. 



The sample design sought a combined Pell, Campus-Based, and GSL sample 
of 3,200 students; however, actual sampling yielded 2,996 student cases. 
After the two major phases of data collection (parent and student 
interviews and student record abstraction) 2,472 cases contained data 
sufficient to calculate fully the various error measures. All findings 
were adjusted for nonresponr and weighted to permit program-wide 
estimates. The interview response rate for all sampled students and 
parents was 87.3 percent. National data estimates are generally within 10 
percent, yet some variation exists. The actual coefficients of variation 
are contained in Procedures and Methods , a separate volume of this report. 
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2,4 DATA PREPARATION AND PROCESSING 



All data collection instruments were received in hard copy form. The 
preparation and processing of data collected in the field by both Advanced 
Technology and Westat required a numoer of steps and operations in order 
to produce a set of clean data tapes ready for analysis. The data were 
subjected to a series of stringent quality control checks both before and 
after keypunching. 



Once the files passed these tests, and were determined to be clean and 
ready to be used in the analysis. Advanced Technology performed ths 
necessary merging of these data files to produce a complete master file 
for Stage Two. 

Analysis of data to identify and measure error involved numerous 
steps. The first step was selecting the most relevant or reliable data 
source or "best value** for each student/ parent/ and institutional data 
item required for recomputation of awards and analysis. 

The multiple data sources used in the study mean that many different 

values emerged during the course of checking on applicat'^n values of 

students and parents. If these values were consistent/ best value 

selection was a simple matter. If these values differed/ however/ a 



method was required to determine the best value. The best value was the 
one that was documented and came from the most reliable source. This was 
determined by merging the data from the various sources and selecting the 
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% best value using a computer program designed for that purpose. In all 

I 

>^ cases, however/ the program defaulted to the value reported by the 

I applicant if more reliable data were not available. Numerous internal 

^ quality control checks were implemented to ensure the accuracy of the data 

I and the procedures used to make the error estimates presented in this 

\ report. 



; Many different types of analyses were performed for the study. These 

; included descriptive tables, cross-tabulations, chi-square analyses/ 

;^ frequency distributions/ and logistic regression analyses. Where 

I 

appropriate we have made comparisons between this study and previous 
quality control studies. 
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FINDINGS OF ERROR 



This chapter presents a summary of the indinqs report for the Stage 
Two study. The data presented in this chapter, along with the more 
detailed data presented in Findings , show the magnitude and sources of 
error in the Title IV programs. Several levels of error are presented, 
from the most aggregated data at the program level to detailed data by 
student application item and institutional procedure. We also present 
data in this chapter that summarize our findings of error with respect to 
characteristics that were associated with ^cudent and institutional 
error, as well as institutional quality control procedures and validation. 

3.1 ERROR IN THE PELL PROGRAM 

The Stage Two study found error in the Fell program to be reduced 
from 1982-83, but still high at all levels. Our estimates show that 
absolute overall error averages $502 per recipient with error, and totals 
$763 million, or 21 percent of total program funds awarded. Fifty-four 
percent of Pell recipients have either been given too large or too small 
a grant. In composite, nationwide these errors totalled $585 million in 
overawards and $178 million in underawards. These data are summarized in 
Exhibit 3-1. 

Error estimates for Stage Two and for Pell Stage Three are presented 
in Table 3-1. In both studies, there were approximately 50 percent of 
Pell recipients with error. The percentage of program funds awarded in 
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EXHIBIT 3-1. A SUMMARY OF AMOUNTS AND RATES OF ERROR 
IN THE PELL GRANT PROGRAM, 1985-8« 
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TABLE 3-1 

APPROPRIATE COMPARISCttiS BETWEEN THE 1982-83 
and 1985-86 PELL GRANT PROGRAM ERROR ESTIMATES* 



Academic Year 1982-83^ 
Academic Year 1985-86^ 



Program- 
wide Error 
($ Millions) 

591(605**) 

763 



Percentage 
of Dollars 
Awarded 

25 (25**) 

21 



Percentage of 
Recipients with 
Error 

49-7 (62.7**) 

54.4 



1 
2 



Using a + $50 tolerance 

These are error estimates using a + $2 tolerance as originally done 
in the Pell Stage Three study* 

Amount: of Pell awards was $2*4 billion for 1982-83 
Amount of Pell awards was $3*6 billion for 1985-86 



error were also very similar; approximately one fourth were awarded in 
error in each of the studies* 

The figures in Exhibit 3-1 indicate that payment error in the Pell 
Grant program affects approximately one-half of all students, with 
students contributing a larger share of error than institutions. These 
results are similar to the 1982-83 study, but represent a slight increase 
in the percentage of students with Pell Grant payment error. 

In terms of program-wide error, there has been a slight positive 
change between the studies. The percentage of program funds awarded in 
error is 21 percent for the current study, a small decrease (4 percentage 
points) from the Pell Stage Three study. The percentage of recipients 
with error also remained similar - approximately one-half of Pell 
recipients have error for both the current and previous studies. 

As part of the Title IV study we analyzed the marginal effects of 
student application itens on Pell payment error. Exhibit 3-2 graphically 
displays the highest student errors by net payment error. Our analysis 
indicated the following: 

• Other nontaxable income errors occurred in 7.1 percent of the 
Pell cases, accounting for $75.1 million in net error. 

• Errors in home equity were slightly less frequent (6.5 
percent of these cases) and caused $64 million in net payment 
error. 

• While relatively infrequent, errors in adjusted gross income 
(AGI) were present in 3.4 percent of the Pell cases and 
accounted for $20.6 million in net error. 
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A useful measure in comparing the relative magnitude of errors 
between 1982-83 and 1985-86, because of the many changes that have 
transpired in the Pell program during the 3 years betwe^en Title IV Stage 
Two and Pell Stage Three, is ranks of net marginal error associated with 
application items. Table 3-2 shows that of the top 10 marginal errors 
for 1985-86/ all were ranked in the top 11 in 1982-83. This shows that 
the application items which cause the highest net marginal errors in 
1985-86 also caused the h/ghest marginal errors in 1982-83 in spite of ED 
implemented corrective actions. 



Pell institutional error can be separated into four components: 
enrollment status, cost of attendance, calculation, and categorical 
error. Exhibit 3-3 displays the highest institutional errors by net 
payment error. Our analysis of marginal institutional error indicated: 

♦ Enrollment status errors were the most frequent, occurring in 
18.2 percent of the cases, and accounted for $9.6 million in 
net error, but consisting of $110.5 million in overawards and 
$100.9 million in underawards. 

♦ Cost of attendance error and calculation error occurred in 
7.3 percent and 7.7 percent of the cases respectively. Cost 
of attendance errors totaled $8.3 million net, and 
calculation errors $3.7 million. 

♦ Generally, categorical errors are an administrative problem 
that can be solved by obtaining additional documentation 
( financial aid transcripts, etc. ) . Most student award 
amounts will not change when this error is corrected (except 
awards to students who have a bachelor's degree). 
Nonetheless, these errors are a regulatory violation that 
must be corrected, although not of the same severity as some 
of the other types of error. 

♦ Categorical errors (errors which cause a recipient to be 
categorically ineligible) occurred in only 4.1 percent of the 
cases, but accounted for $114.2 million in institutional 
payment error, all overawards. This is due to the fact that 
a categorical error makes the student's entire award an over- 
award. 



TABLE 3-2 

A COMPARISON OF RANKS OF THE IMPACT Off 
NET PROGRAM-WIDE PELL STUDENT ERROR FOR 
INDIVIDUAL APPLICATKM ITEMS 
1982-83 AND 1985-86 



Item 



Ranks 

1985-86 1982-83 



Other Nontaxable Income 

Home Equity 

Dependency Status 

Dependent Student's Assets 

Student's Expected Taxable Income 

Household Size 

Adjusted Gross Income 

Number in College 

Dependent Student's Income 

Investment Equity 

AFDC 

Dependent's Nontaxable Income 

Business/Farm Equity 

Cash/Checking/ Savings 

Student's Expected Nontaxable Income 

Educational VA Benefits 

Dependent's Taxes Paid 

Elementary and Secondary Tuition 

Parent's Marital Status 

Student's Spouse's Expected Income 

Student's Marital Status 

Mother's/Spouse's Earned Income 

Medical Expenses 

Father's/Student's Earned Income 

Social Security Benefits 

Federal Taxes Paid 



1 
2 
3 
4 
5 
6 
7 
8 
9 
10 
11 
12 
13 
14 
15 
16 
17 
18 
19 
20 
21 
22 
23 
24 
25 
26 



2 
5 
1 
6 

3 
7 
4 
8 
11 
12 

15 
16 

13 

19 
14 

20 
10 
17 
18 
21 
9 
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• The largest of the categorical errors were a missing 
Financial Aid Transcript (2 percent of the cases and $41.2 
million in error)/ a missing Selective Service Compliance 
Statement (1 percent of the cases and $30.5 million in 
error}/ a missing Statement of Educational Purpose (0.7 
percent ^nd $28. 1 million)/ and award to a student with a 
bachelor's degree (0.3 percent and $13.6 million). 

All of these figures show error in the Pell program to be continually 
high/ and a cause for concern. However/ the pattern of institutional 
error is different from student error. Institutional error is more equal 
in terms of overawards and underawardS/ while student error is more 
overawards. This error pattern complicates corrective actions/ and 
suggests that institutional error requires more management controls 
similar to institutional quality control procedures. 

3.2 ERROR IN THE CAMPUS-BASED PRCXSRAMS 

Exhibit 3-4 provides a summary of need error in the Campus-Based 
programs. Need error is a measure of the diffeience in need using best 
and reported data and does not necessarily translate into award error. 
(Error data on awards in excess of need and distributional errors are 
provided in detail in Findings . ) The results indicate that absolute need 
error averaged $1,080 per recipient with error, and occurred in 77 
percent of the cases. Overstatements of need are more prevalent than 
understatements of need, and result in $786 million in program-wide 
error/ while understatements account for $282 million in program-wide 
need error. These are comparable? to results from Stage One. 
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For instance, in the pilot phase (Stage One) approximately 70 percent 
of Campus-Based recipients had need error that totaled $978 million 
(absolute). The current figures are comparable to these results, and are 
the first true estimates of error in the Campus-Based programs based on 
the methodology tested in the pilot phase of this study. The slight 
increase in error in Stage Two may be attributed to an 
underrepresentation of error in Stage One due to changes in the study 
methodology. 

Need error is important because it represents the degree to which 
financial aid packages are based on incorrect data. Some need error is 
so large that awards are made, erroneously, in excess of true need. In 
fact, awards in excess of true need were made for 22.5 percent of 
Campus-Based aid recipients and totaled an estimated $265 million. 

Exhibit 3-4 decomposes Campus-Based need error into student and 
institutional errors. Student need error affects more than twice as many 
recipients as institutional need error (65 percent versus 32 percent). 
Mean error per recipient with error is $1,012 for student error and $877 
for institutional error. Again, student error is much larger than 
institutional error. Program-wide error attributable to student error 
was over two times as high as institutional ($853 million versus $353 
million). Clearly, student need error contributes a much larger 
percentage of absolute need error than institutional need error. 
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EXHIBIT 3-4. A SUMMARY OF AMOUNTS AND RATES OF NEED ERROR 
IN THE CAMPUS-BASED PROGRAMS, 1985-86 



41 



Marginal student need error measures the effects of misreporting of 
application items on student need error. The marginal need error for an 
application item measures the need error removed if that item were 
independently verified. Our analyses indicated the following: 

• Students* expected taxable income errors occurred in 16.1 
percent of Campus-Based recipients and caused $114.8 million 
in net need error. This was followed by student's expected 
non-taxable income (8.9 percent, $114 million). 

• Household size and number in college errors were also large. 
They occurred in 12.6 and 7.2 percent of the cases and 
accounted for $58.7 million and $26.5 inillion in net need 
error. 

• AGI has a net marginal need error of $8.5 million in 
understatements. The absolute marginal need error for AGI is 
$74.7 million. 



Campus-Based institutional error can be separated into seven 

components: factoring Pell awards, cost of attendance, EFC error, 

factoring GSL awards, initial overawards, disbursement, and categorical 
error. Our analyses indicated the following: 

• Errors in factoring Pell awards (errors in including the 
correct amount of a students* Pell grant or estimated grant 
when calculating Campus-Based aid) occur the most frequently, 
almost 20 percentage points more than any of the othei errors. 

• Errors in factoring GSL awards were also significant. While 
these errors occurred in only 1.1 percent of Campus-Based 
cases, they caused $16.9 million in net need error. 



3.3 ERROR IN THE GSL PROGRAM 



GSL certification error occurs whenever the amount certified exceeds 
the difference between cost of attendance and the resources available to 
meet these expenses. Available resources would include known aid from 
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Pell, Campus-Based, and other programs and expected family contribution 
for students with adjusted gross incomes over $30,000. (The study was 
conducted prior to changes in the GSL program, which eliminated the 
exemption from need analysis for recipients with income of $30,000 or 
less.) Exhibit 3-5 presents a summary of overall GSL certification error. 

Error estimates from Stage One in the GSL program are considerably 
different from estimates in the current study. There was a total of $441 
million in certification error which averaged $1,215 for the 11 percent 
of the cases in error in Stage One. However, a significant 
methodological change was implemented for Stage Two; Expected Family 
Contribution (EFC) error was measured in Stage Two and had a significant 
impact on certification error. 

In the GSL program, institutional certification error is higher for 
all measures than student certification error. Institutional error 
affects 14 percent of the certifications with error, while student error 
affects 11 percent. Institutional error accounts for $587 million in 
prograin-wide error; approximately one-third more than student error which 
accounts for $393 million in program-wide error. The mean error per 
certification with error was also about $175 larger per certification 
with error for institutional error ($1,238) than student error ($1,065). 

The relatively hicih proportion of institution versus student error is 
probably due to the fact that application items used for the Pell and 
Campus-Based programs do not affect GSL certifications for applicants 
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Due to revision in the estimate of total 6SL loan volume, ttiese figures should 
be reduced by approximately 10 percent 

Error Is defined as a discrepancy of $50 from the best award. 



EXHIBIT 3-5. A SUMMARY OF AMOUNTS AND RATES OF ERROR 
IN THE GSL PROGRAM, 1985-86 
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whose family AGI is $30,000 or less. For applicants reporting $30,000 or 
less in family AGI, significant student error can occur only if che 
actual family AGI is greater than $30,000. Family AGI and dependency 
status are the most meaningful application items to associate with 
student certification error . However, we did calculate marginal student 
errors even though there was no overcertif ication consequence in many 
cases. Our findings indicated: 

• Among students who claimed to be independent, the student 
certification error rate was over five times as high for 
those who were actually dependent. In addition, the mean 
error was over twice as high for these students. 

• Of the students who reported $30,000 or less in family AGI, 
just over 2 percent had family AGI in excess of $30,000. 
These students had a certification error 64 percent of the 
time. 



GSL institutional certification error can be classified into seven 
mutually exclusive categories: factoring Pell awards, cost of 
attendance, EFC error, factoring Campus-Based awards, factoring other 
aid, initial overawards, and categorical error. Our analyses indicated 
the following: 

• Errors made by institutions in computir EFC had the largest 
impact on certification error. Three possib e areas where 
institutions can make mistakes in computing the EFC to use in 
the GSL program are: use of an EFC different from the EFC 
used in awarding Campus-Based aid, failure to use the GSL 
Tables properly, or incorrect determination of whether family 
adjusted gross income is over or under $30,000. 

• Failure to use the GSL Tables had the largest impact on EFC 
error. In almost one-fourth of the cases where the GSL 
Tableswere used to calculate EFC, there was a certification 
error . 

• Categorical errors were present in 1.3 percent of the 
certifications, and totaled $142.8 million. 
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• Initial overawards were a significant source of institutional 
error/ occurring in 3.3 percent of the cases and $130 million 
in overcertif ications. 

• Errors in factoring other aid were present 2,2 percent of the 
time and caused $41.5 million in overcertif ications . 

In conjunction with errors in factoring other aid/ initial overawards 
indicates institutional problems in identifying and coordinating all 
sources of aid received. Errors in factoring Pell awards and 
Campus-Based aid demonstrate the interaction between programs where 
errors in one program affejt the eligibility for another. Errors in 
factoring Pell awards wc^re a problem in GSL just as they were for the 
Campus-Based programs. 

3.4 ANALYSIS OF ERRORS IN THE TITLE IV PROGRAMS 

We analyzed the relationships between student and institutional 
charactei:isti::s and the prevalence of aggregate error in the Pell/ 
Campus-Based/ and GSL programs. We tried to identify characteristics 
that were associated with error so that we could develop corrective 
actions. Our initial analyses indicated several characteristics of 
students that were associated with either increased or decreased student 
error in one or more of the programs. In addition/ we found a group of 
institutional characteristics to be significantly associated with student 
or Institutional error in one or more of the programs. 

We further analyzed these characteristics using multivariate 
techniques to develop corrective actions. After using these multivariate 



techniques to control for other variables/ several of the characteristics 
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were no longer significant in some of the programs for which they were 
previously determined to be significant* None of the characteristics 
remained significant across all programs. 



These analyses suggest that error occurs in very complex, 
intertwined, and perhaps even contradictory patterns. There is no 
one-dimensional profile of the error-prone student or the error-prone 
institution. This is one reason why mechanical, quick-fix corrective 
actions remove only a portion of error, but are not able to reduce the 
more deeply imbedded causes of error. 

3.5 VALIDATION AND INSTITUTIONAL QUALITY CCMTTROL 

This section presents the analysis and conclusions concerning the 
extent and effectiveness of two primary dimensions of quality control in 
the Title IV programs: validation of student application data and 
institutions* quality control procedures used to control institutional 
error in the student aid programs. Analysis of validation focused on the 
extent of activities and the effectiveness of validation in removing 
error among Title IV recipients selected for validation by the Pell 
Processor, those selected by institutions, and compares these recipients 
with those not selected for validation for each Title XV program. 

We found validation in the Pell program to be quite extensive. Our 
findings indicated: 

• Validation activities in the Pell program occurred for 80 



percent of all Pell recipients. 
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• students selected for validation by the Pell Processor had 
the highest rates of error on their initial applications for 
the six data items mandated for validation. After validation 
the remaining error in these items was not much different for 
Pell selected/ institution selected/ and non-selected 
students . 

• The Pell Processor does relatively well selecting recipients 
prone to making errors on the six data items required to be 
validated. However, the discrepancies found in these six 
data items are not the predominant contributors toward 
payment error. Neither the Pell Processor nor institutions 
do well in selecting students prone to making errors in the 
other application items. As such/ neither do well 
identifying applications that result in higher than average 
absolute payment error. This ii* especially true of 
applications that result in overawards. 

• Institutions do a good job of removing potential error 
through validption. Recipients with potential payment error 
after their first transaction had reductions in potential 
error of $164 per recipient/ or $85 million/ for Pell 
selected and $125 per recipient/ or $43 million/ for 
institution selected recipients. 

• After validation/ error still remains high. Pell selected 
recipients had $161 error per student/ institution selected 
recipients had $127 error per student/ and non-selected 
students had $207 error per student by the time final awards 
were made. ThuS/ while effective in removing some level of 
error/ corrective actions in addition to validation are 
needed if there is going to be substantial progress made in 
reducing error in the Pell Grant program. 

Findings concerning validation in the Campus-Based programs were 
*:imilar to those in the Pell program. Our findings indicated: 

• Validation activities in the Campus-Based programs were about 
as extensive as in the Pell program (73 percent) and 
increased between 1983-84 and 1985-86 in terms of both 
recipients selected by the Pell processor and those selected 
by institutions. 

• Recipients receiving aid from multiple programs (e.g.. Pell 
and Campus-Based) were more likely to be selected for 
validation by institutions than were recipients of only 
Campus-Based awards. 
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• Campus-Based recipients selected by the Pell Processor for 
validation have the lowest rates of student need error (55.2 
percent) . 

• All validated cases have lower item discrepancy for adjusted 
gross income/ although institution selected validation is not 
successful at reducing item discrepancy in general. 

Finally/ our findings* concerning validation in the GSL prograr.i were 
noticeably different than the Pell and Campus-Based programs. Validation 
in the GSL program was relatively rare, and in general/ our findings 
indicated: 

• Institutions selected Campus-Based and GSL rec ipients for 
validation at a higher rate (67 . 9 percent) than recipients 
receiving only a GSL (45.5 percent). 

• Institution selected GSL recipients had an estimated 34.5 
percent overcert i f ication error rate as compared to 28.7 
percenL and 27.7 percent error rates for Pel] selected and 
not selected respectively. 

Institutional Quality Control Analyses 

We analyzed the extent and effectiveness of institutional quality 
control (QC) procedures/ to assess what qudlity control procedures 
institutions used/ how often the procedures were employed/ and what 
follow-up procedures an institution used after correcting an identified 
problem. 

The use of QC procedures vai. led a great deal across institutions by 
type and control. Both 2-year public institutions and proprietary 
institutions relied upon manual or auxiliary QC procedures (e.g./ 
checking with other offices/ interviewing students/ using internal audits 
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and auditors^ and using consultants) while 4-year public schools made 
predominant use of automated and sampling procedures (selecting students 
et random/ ano using the sample to confirm the validity of specific data 
items)* Two year, and 4-year private schools predominantly used sampling 
and sampling in conjunction with either manual QC procedures or other 
auxiliary QC procedures^ respectively* 

In the Pell Grant program, recipients attending institutions that 
used QC procedures that involved sampling had the lowest institutional 
error rates (19 percent), while recipients at institutions with little or 
no QC had the highest institutional error rates (38.5 percent). 
Recipients at schools using automated and sampling procedures had the 
lowest rate of institutional Campus-Based need error. Recipients 
attending schools wi«-h little or no QC had the highest rates of 
institutional Campus-Based need error. One minor difference from the 
results in the Pell program is that recipients at institutions using 
mixed types of QC followed those in the automated/sample category in 
having the lowest rate of institution Campus-Based need error. 

In general/ these findings indicate that error in the Title IV 
programs continue to be high. The fact that validation and institutional 
quality control procedures were associated with lower error rates in many 
cases suggests that these are viable options that should be explored 
further for possible corrective actions. 
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4*0 

CORRECTIVE ACTIONS 



The ultimate purpose of the Department of Education's (ED*s) quality 
control studies is to reduce error and improve the quality of the Title 
IV programs. An important component in ED's quality improvement strategy 
is the analysis and implementation of corrective actions* Tliis section 
reports on the corrective actions analyzed in the Title IV QC study. 

4.1 CORRECTIVE hCTlC^S FRAMEWORK AND STRATEGY 

As part of its corrective actions strategy, ED has conducted numerous 
quality control studies of the student financial aid programs. 
Consistently, these studies have found quality problems to be a major 
concern for each of the programs. Because error has been pervasive at 
all levels (item-level as well as system-wide), it is necessary to 
examine four different levels of corrective actions. These four levels 
are corrective actions aimed at errors in individual data items or 
individual c^^moonents of the delivery systems, corrective actions for 
groups or classes of similar data items or components of the delivery 
systems, corrective actions based on changes in strategic appica'^^es to 
improving quality, and corrective actions aimed at major problems ±n the 
programs that involve structural changes in the delivery system. 
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Previously each of these types of corrective actions has been 

determined to have merit. Therefore, the current study and investigation 

of corrective actions examined corrective actions at each level. These 
levels are differentiated as follows: 



• Level I corrective actions are designed in response to 
significant errors in individual data items or individual 
components of the delivery system, often as short-term 
measures. Many Level I corrective actions have been 
undertaken. (These include changes in the items included in 
validation activities anc' increasing the proportion of 
recipients selected for validation.) Additional actions are 
possible, but will not bring down error rates in d lasting 
and cost-effective manner. These corrective actions rely on 
costly, after-the-fact inspection methods, and Congress has 
constrained ED in these areas. Validation is occurring in 80 
per cent of Pell Grant cases already. Therefore little more 
can be expected through added validation. Nor would changes 
in procedures used to validate data be likely to have a major 
impact on error reduction. Analysis of various validation 
procedures did not indica*"e differences in their ability to 
remove error. 

• Level II corrective actions are those oriented towards groups 
or clas<^es of data items, or types of components of the 
deliver^' system. Many Level II corre'' live actions have also 
been undertaken. V/hile additional Level II corrective 
actions can be undertaken, the nature of them makes them 
costly and of questionable effectiveness in the long run. 

• Level III corrective actions are those that constitute a 
shift in the approach to quality. Some Level III corrective 
actions are under investigation, but management decisions on 
full-scale implementation or expansion are needed. 

• Level XV corrective actions are those that are longer-term 
and involve major, in many cases structural, changes in the 
delivery systems or the Title IV programs. At this point. 
Level IV corrective actions have not been targeted for 
implementation, and there is no long-term plan in place for 
developing them. 

By their very nature. Level I and Level II corrective actions tend to be 
oriented towards liability-type errors, while Level III and Level IV 
corrective actions focus on aspects that transcend liability errors and 
focus more on the overall quality of the delivery system. 
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ED faces a decision whether to maintain the status quo or to mobilize 
for cJ'.ange to significantly improve the quality of the Title IV delivery 
systems* ED must either live with the current levels and rates of error, 
since most mechanical fixes have been exhausted, or take bold steps to 
restructure the delivery system itself, including focusing on 
results-oriented procedures rather than prescriptive, process-oriented 
ones . 

4.2 CORRECTIVE ACTIONS FOR SIGNIFICANT STUDENT AND INSTITUTIONAL ERRORS 

The corrective actions presented in this section fall in the Level I 
category and are aimed at individual student application data items and 
institutional procedures. 

A significant percentage of misreporting of home equity, savings, 
dependent student's assets, and other non-taxable income was found to be 
due to erroneously reporting a 2ero value for these data items. Of the 
applicants who reported 2ero for the following items, those who reported 
a zero value incorrectly are as follows: 



Parent's home equity 



15.4% 



Independent student's home equity 



6.5% 



Parentis savings 



37.3% 



Independent student's savings 



28.7% 



Dependent student's assets 



31.3% 



Other non-taxcX)le income 



21.6% 



The problem of erroneously reporting zero is occurring in items for 
which values could be cross-checked on the Federal tax form filed by the 
applicant or his/her parent(s). While none of the values of these items 
can be obtained directly from the tax return, values on the tax return 
can indicate situations where a data item exists when none was reported 
on the application. Therefore, using the Federal tax form as a source of 
information to determine when values should be verified as opposed to 
using the form for verifying the amount of the values — should be 
investigated by ED. 

In addition, changes in application forms and clarification of 
instruction!, could improve the accuracy of several data items, including 
other non-taxable income, household size, and number in college. 

Our analyses indicated that the majority of errors in household size 
and number in college do not occur because unforseen circumstances cause 
students' estimates of these values to change after they apply for aid. 
Of the recipients whose reported household size was less than their best 
household size, 70.4 percent did not have an unanticipated change in 
their household size. Of those whose reported household size was greater 
than best, 60.4 percent had no unanticipated change. Similar figures 
exist for number in college. Of those recipients whose reported number 
in college was less than best, 77.4 percent had no unanticipated change, 
and 59 percent of those recipients whose reported number in college was 
greater than best had no unanticipated change. 




Enrollment status errors in the Pell program suggest that 
institutions have problems calculating enrollment status correctly for 
non-standard students. Institutions seem to have trouble adjusting a 
student's enrollment status for summer sessions, and making adjustments 
when other changes in enrollment status occur. Also problematic is 
calculating enrollment status for clock-hour students. These factors 
suggest that ED may want to issue clarifications concerning determining 
enrollment status for non-standard students. In addition, ED may want to 
investigate changing the procedures for determining, and adjusting, 
enrollment status during summer sessions and for clock-hour students. 

A subset of procedural or calculation errors appeared to be 
disproportionally distributed at a few institutions. For these 
institutions, Sdinpling a relatively small group of recipients would 
uncover the occurrence of these systematic problems. Therefore, ED could 
investigate the characteristics of these institutions by drawing 
institution samples. ED may wish to design a sample in its audits of 
institutions taat would indicate if institutions are having systematic 
problems with these items. Alternatively, ED could issue technical 
assistance or clarification concerning the problems in these items either 
through professional associations or through its own channels. 

4.3 PROSPECTIVE INCOME DATA 

Two Level II corrective actions were analyzed as part of this study. 
The first of these was an analysis of the effects of redefining 
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the group of prospective income data items. Recent changes rr.ade by 



Congress in the Title IV programs do not allow for prospective income 
data to be used in determining either awards in the Pell Grant program or 
need in the Campus-Based programs, except for dislocated workers. Our 
analyses of the likely effects of this change in the Pell and 
Campus-Based programs produced the following findings: 

♦ The distribution of Campus-Based need using base year income 
data will be substantially different than the distribution 
using prospective income data. 

♦ The move to base year income data will cause a decrease in 
total need in the Campus-Based programs. 

♦ Independent students' need error rate in the Campus-Based 
programs, attributable to income, using base year data are 
less than half of the error rates using prospective data 
(24.7 percent vs. 66.2 percent). 

♦ Fewer than 20 percent of Pell recipients are affected by the 
change to base year income data (not including Special 
Condition filers) . 

d The current method of selecting which Pell recipients should 
have their awards based on prospective income is in error 
approximately 29 percent of the time (15.7 percent of Pell 
recipients use prospective income when they should not and 
13.1 percent do not use it when they should). 

♦ The change affects Pell recipients with low awards the most. 
Nearly half of those becoming ineligible had awards of less 
than $750. 

♦ Dependent students* error rate in the Pell program, 
attributable to income, using base year data declines 
approximately 25 percent (from 16 to 12 percent). 

Because error decreases substantially when using base year data and 
therefore aid is more likely to go to those who truly need it, this 
change is likely to be judged favorably in spite of causing a decrease in 
total need for Campuw -Based funds. 
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4.4 REDEFINED DEPENDENCY STATUS 



The second Leve] II corrective action analyzed was the changes made 
in the data items used to determine dependency status. Beginning in the 
1987-88 acadcr.iic year, a redefined set of data elements and rules will be 
used to determine the dependency status of applicants for Title IV aid. 
We analyzed this change to determine the likely impact of the redefined 
regulations and to identify ways in which 'he rules could be modified to 
achieve improved results • 

• In the aggregate, there is no change in the distribution of 
independents and dependents between the current and redefined 
dependency status regulation* An estimated 14 percent of 
recipients change dependency status under the redefined 
model, 7 percent going from independent to dependent and 7 
percent from dependent to independent. Recipients who are 22 
or 23 years old are the most likely to change dependency 
status under the new definition. 

• The percentage of recipients reporting as independents who 
should have been dependents is lower under the new 
definition, but the percentage of recipients reporting as 
dependents who should have been independent is higher. Thus 
the error rate of dependency status error is not greatly 
affected by the new definition* 

• We recommend that the self-sufficiency criteria be expanded 
in the new definition (i*e*, all or most students should be 
required to meet the self-sufficiency criteria). This will 
help minimize the effect of students changing dependency 
status simply because of the new definition. Modifying the 
new definition to expand the scope of the self-sufficiency 
criteria greatly reduces the problem of recipients who are 
dependent under the current model becoming independent under 
the redefined model. The modification does, however, cause a 
number of current independents to become dependent under the 
redefined model* 
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4.5 INSTITUTIOTAL QUALITY CONTROL AND VALIDATION 



A key component of the corrective action analyses and reconunendations 
were our analyses of institutional quality control procedures and 
validation of student data. These are analyses of Level III corrective 
actions. We analyzed data in these areas to determine what types of QC 
procedures were most effective in controlling institutional error, and if 
any validation-related procedures were associated with increased 
effectiveness of validation. Our analyses indicated the following: 



Institutional Quality Control Procedures 

• Institutional quality control procedures were generally 
associated with lower rates of institutional error in the 
Pell and Campus-Based programs. 

• Sampling-based QC procedures were associated with lower rates 
ot* error more often than other QC procedures. Stuctents at 
institutions that used sampling-based QC procedures had 
institutional error 19 percent of the time, while students at 
institutions with little or no QC had institutional errors 
38.5 percent of the time. 

• Higher levels of either professional or clerical/data entry 
staff and higher levels of automation were, for the most 
part, not significantly associated with lower rates of 
institutional error. 

• These findings support continued development of the 
Institutional Quality Control Pilot Project and similar 
activities. In addition to the Pilot, materials and 
information concerning qual ity control procedures could be 
developed as technical assistance materials for institutions 
not participating in tae Pilot. 
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Validation 



• Pell selected cases have a higher probability of having a 
student error removed than not selected cases. 

• Institutional selected cases also have a higher probability 
of having errors removed than not selected cases. This 
difference, however / is not statistically significant (i.e./ 
it could have occurred due to sampling error). 

• Validation-related procedures that could be used in designing 
corrective actions were not significant in explaining 
differences in error removed through validation. 

As we have previously stated/ about 80 percent of Pell Grant 
recipients are already undergoing some type of validation. Further 
improvements in error reduction will therefore not come from validating 
more students. Additionally/ there do not appear to be better 
validation-related procedures that could be recommended to improve the 
institutions* ability to reduce error. Therefore, the most likely avenue 
for imptcvoment in val.?iat.ion is through better techniques for 'aigeting 
applications and data items for validation. 

4,6 SIMPLIFICATION OF THE DELIVERY SYSTEM 

The findings from numerous quality control studies, and a pervasive 
perception that student aid in general, and the formulae used to 
determine eligibility and need for Federal financial aid funds in 
particular, are too complex, has led to a widespread interest in 
simplifying these formulae. In Stage One of the current study/ EO 
conducted a multifaceted assessment of Pell simplification which focused 
on equity and quality issues. 




Simplification involves many aspects other than just the formulae. 
Also included are simplifying the structure of the delivery system 
itself, as well as changing the focus of quality assurance activities 
from process-oriented activities to results-oriented requirements. 
Targets n'or simplification include deregulation, focusing on collections 
and default rates rather than procedural requirements, redirecting due 
diligence procedures to results and allowing schools to develop their own 
methods, and integrating more procedures and regulations across all of 
the Title IV programs. 

For this study, we analyzed the effects of reducing the number of 
data elements used in the Uniform Methodology (UM) formula. The Uli 
formula is used to determine a student's Expected Family Contribution 
(EFC) in the Campus-Based programs. For our analyses we truncated the UM 
formula to six items: dependency status, AGI, U.S. taxes, other 
non-taxable income, household size, and number in college. This 
truncation was based on the same criteria used in developing a reduced 
data element formula for Pell in Stage One, namely budget impact, 
aggregate distributional impact, sensitivity, reliability, and 
verif iability . Our analyses indicated the following: 

• The distributions of need in the Campus-Based programs and 
certification in the GSL programs resulting from the reduced 
formula closeiy approximate the intended distributions . 
(Intended distributions are those that would have occurred 
had accurate data and procedures been used to distribute aid 
under current formulae and regulations). 

• Most recipients have minimal changes in their Campus-Based 
need and GSL certification under the reduced formula. 




♦ student error in the Campus-Based and GSL programs is 
significantly lowei.' under the reduced formula, 

• Adjusting aspects of the truncated formula (e.g., including 
additional items, eliminating some/ changing how the items 
are used in the formula, etc.) could improve on the accuracy 
and distribution of the reduced formula. 

Shortening the UM formula shows promise in increasing the accuracy 
and efficiency of determining need in the Campus-Based programs. The 
distribution of need under a reduced formula overcomes to a degree the 
distortions from the intended distribution caused by student reporting 
error. Equity is improved by removing errors that undercut that very 
objective. Thus, simplification will help ED achieve the objectives of 
the programs through a delivery system that is less vulnerable to frauc 
waste, and abuse and without undercutting the distribution of aid to the 
most needy recipients. The distribution of need under the reduced 
formula appears especially attractive since the target group of 
re^cipients, those in the lowest income groups for independent and 
dependent recipients, gain slightly in amount of need, relative to the 
other income groups. 

Furthermore, because the amount of need recipients have under the two 
models is nearly the same over 60 percent of time, and the amount of 
student need error under the reduced formula decreases dramatically, the 
possibility of the reduced formula achieving ics goals is significant. 
The ultimate reduced formula might not take the same form or have the 
same data elements as the reduced formula used for these analyses. 
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The effects on the GSL program suggest that reducing the UM formula 
has promise in increasing the accuracy of determining certifications in 
the GSL program too. The distribution of certifications under the 
reduced formula approximated the intended distribution very closely. In 
addition, the effects on certification for both graduate and 
undergraduate students is such that most students receive nearly the same 
certification under the reduced formula. Finally, GSL student error 
under the reduced formula is significantly lower than under the full 
formula. 

4,7 CONCLUSIONS 

The results of analyses of findings and corrective actions indicate 
that ED now faces a critical decision in improving the quality of the 
Title IV delivery system. Error continues to be high in spite of 
corrective actions already taken. Yet the corrective actions ED has 
taken have nearly exhausted the options for using mechanical approaches 
to reducing error in individual data items. ED must either accept erro^ 
rates of the magnitude that currently exist, by relying on costly 
after-the-fact inspection techniques, or accept the challenge of 
restructuring the delivery system itself to design error out of the 
process. In order to do this, ED must establish goals and targets as 
part of a larger quality improvement strategic plan including the 
prevention of error through results-oriented requirements. Two other 
major analyses are being conducted as part of this long-term plan. One 
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is embodied in a paper, developed under this contract/ entitled Delivery 
System Quality Improvements * The second is a study which will address 
lender and guarantee agency quality, and propose corrective actions/ 
which will soon be issued. 

This study and previous studies have shown that some of the very data 
on which billions of taxpayer dollars are being disbursed are vulnerable 
to error/ and can only be improved through redesign of the delivery 
system. This approach will require clear commitment from all interested 
parties and internal management. Unless ED is willing to accept error 
rates and magnitudes of the current level/ redesign of the Title IV 
delivery system must be undertaken. 
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